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OPTIMIZATION  IN  OPERATIONS  RESEARCH:  SOME  EXAMPLES 


by 

George  B.  Dantzig 

Mathematical  methods  are  penetrating  into  everyday  decision  processes. 
Historically,  planning  has  always  been  carried  out  using  what  some  call 
"mature  Judgment".  The  human  mind  has  a  remarkable  faculty  for  trying 
to  consider  many  possible  alternatives.  The  electronic  computer  with 
its  equally  remarkable  speed,  appears  to  be  a  natural  tool  for  exploring 
alternative  courses  of  action.  It  takes  little  imagination  to  see  how 
computers  can  be  programmed  in  the  finest  minutia  to  represent,  say,  the 
quantities  of  inventory  on  hand  and  the  movement  of  in-process  inventory 
from  machine  to  machine  in  a  Job-shop.  This  obvious  approach,  called 
simulation,  has  popular  appeal.  Every  manager,  aware  that  he  must  keep 
up  with  the  times,  can  easily  imagine  a  computer  sifting  through  the 
possible  alternative  combinations  and  selecting  one  that  is  best  according 
to  some  criterion.  In  fact,  the  less  such  a  person  knows  about  mathematics, 
the  more  this  simulation  approach  has  appeal. 

However,  a  raw  search  often  can  be  combinatorially  prohibitive. 
Consider,  for  example,  a  problem  represented  by  a  linear  program.  Math¬ 
ematically,  the  set  of  possible  feasible  programs  is  represented  by  the 
intersection  of  a  number  of  half-spaces  whose  boundaries  are  hyperplanes. 
The  optimal  solution  consists  in  finding  that  vertex  which  is  the 
greatest  distance  from  a  given  object ive-hyper-plane  'the  dotted  one  in 
Figure  l).  The  set  of  "feasible"  solutions,  (the  shaded  area)  can  be 
represented,  of  course,  by  linear  combinations  of  vertices  or  extreme  point 
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FIGURE  1 


Problem  1:  In  the  Simplex  Method  the  number  of  vertex  steps  in  practice 

is  of  the  order  of  magnitude  of  M  [the  excess  of  the  number  of  hyper¬ 
planes  over  the  dimension].  Determine  a  sharp  upper  bound  for  the  minimum 
number  of  steps. 
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solution.  Now,  it  is  a  relatively  easy  matter  to  program  a  computer 
to  pass  from  one  extreme  point  to  the  next.  One  approach  therefore, 
is  to  investigate  all  the  extreme  points  and  pick  out  the  best.  Unfor¬ 
tunately,  some  simple  convex  polytopes  like  an  n-cube  have  a  prohibitive 
number  of  vertices  (the  n-cube  has  2n  vertices)  and  this  simple  naive 
vertex-search  approach  is  doomed  to  failure.  Surprisingly  enough, 
a  minor  variation  of  it  has  worked  out  quite  well. 

This  method  moves  from  one  vertex  to  the  next,  moving  along  that 
edge  which  gives  the  greatest  decrease  in  the  objective  function  per 
unit  change  of  the  variable  being  introduced  into  the  solution.  This 
minor  change,  introducing  a  steepest  descent,  would  not  appear  to  be 
sufficient  to  prevent  endless  wandering  over  the  edges  of  the  polytope. 

Yet  all  over  the  world  this  procedure,  called  the  Simplex  Method,  is 
a  work -horse  that  daily  solves  thousands  of  problems.  This  brings  us 
to  the  first  of  several  famous  problems  of  mathematical  programming,  which 
is  stated  below  Figure  1,  In  practice,  the  number  of  steps  is  often  of 
the  magnitude  of  the  excess  m  of  the  number  of  hyper-planes  over  the 
dimension  of  the  space  k  .  The  open  question  is  why? 

The  study  of  the  properties  of  convex  polytopes  forms  an  important 
part  of  linear  programming  research.  It  is  hoped  that  insight  may  be 
provided  on  why  the  simplex  method  is  as  efficient  as  it  is  and  how  it 
might  be  improved.  A  second  unsolved  problem,  closely  related  to  this 
one  is  the  famous  "m-step"  or  "Hirsch  Conjecture"  (see  Figure  2). 

Mathematics  as  we  know  it  today  is  an  eerie  edifice  of  abstraction 
that  has  its  roots  in  problems  of  the  real  world.  But  these  problems, 
aside  from  puz2ft.es  and  number  lore,  find  their  origins  in  the  world  of 
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FIGURE  2 


Problem  2;  Given  a  convex  polytope  in  M  dimensional  space  with 
2M  faces,  M  faces  intersecting  at  0  ,  the  other  M  faces  intersecting 
at  P  : 


Prove  (or  Disprove);  There  exists  a  chain  of  M  edges  connect in. 


physics.  The  every  day  real  world  of  planning  and  decisions  attracted 
slight  interest  among  mathematicians  until  about  19^7.  The  reasons 
for  this  are  obscure.  In  1936  Motzkin,  in  his  thesis,  counted,  after 
exhaustive  search,  some  30-odd  papers  on  linear  inequalities.  [Compare 
this  with  the  number  that  were  published  in  a  single  month  at  that  time 
on  linear  equations  and  the  closely  related  matrix  theory!]  Those  papers 
that  were  published  were  isolated  one  from  the  other.  It  is  strange  that 
there  was  so  little  interest,  for  it  is  easy  to  set  up  mathematical  models 
of  decision  problems  in  the  form  of  linear  inequalities.  Perhaps  it 
might  be  argued  that  the  reason  such  problems  were  never  studied  was 
the  difficulty  of  solution  without  electronic  computers.  But  why  then 
were  equation  systems  studied  when  even  a  20x20  system  was  also  out 
of  the  reach  of  available  computers?  I  do  not  know  the  answer  to  the 
question  as  to  why  inequality  theory  was  not  investigated  more  intensively. 
Today  we  know  that  study  of  inequality  constrained  systems  is  an  inter¬ 
esting  and  exciting  field,  full  of  challenges,  like  any  other  field  of 
mathematics . 

Part  of  the  motivational  drive  comes  from  the  desire  by  industry 
to  solve  bigger  and  bigger  systems.  In  refinery  operations,  an  initial 
simple  blending  model,  will  in  time  be  expanded  until  it  includes  the 
distillation  and  reforming  units,  the  selection  of  crude  oils,  oil  field 
production,  and  the  distribution  and  storage  of  the  multi-product  output. 
Industries  which  make  numerous  products  in  a  number  of  alternative 
plants  and  ship  to  many  destinations,  give  rise  to  very  large  scale  math- 
ematical  models.  One  such  system  has  over  30,000  equations  and  over  10 
non-negative  variables!  In  spite  of  their  size,  this  and  other  such 
problems  have  been  successfully  solved.  The  technique  used  is  called 
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the  decomposition  principle.  It  is  a  special  application  of  the  Lagrange 
Multiplier  approach  which  permits  the  breaking  up  of  the  original  problem 
into  a  sequence  of  smaller  linear  programs. 

Lagrange,  as  you  may  recall,  made  many  contributions  to  mathematics 
from  number  theory  to  analysis.  It  is  said  that  when  the  post  held  by 
Euber  was  vacated,  Frederich  the  Great  invited  Lagrange  with  the  words: 
"from  the  greatest  king  in  Europe  to  the  greatest  mathematician  in  Europe." 
Lagrange,  by  contrast,  was  a  modest  man. 

The  general  approach  is  best  illustrated  by  considering  a  very  broad 
class  of  non-linear  inequality  constrained  extreme  problems.  We  begin 
by  first  considering  classic  equality  constrained  problem. 

Find  x  such  that 

fQ(x)  =  min  x  =  (x1,...,xn) 

fx(x)  =  0 

• 

f  (x)  =  0 
m 

We  form  the  Lagrangian: 

0(x)  =  f  (x)  +  \  f  (x)  +...+  A  f  (x) 

0  1  1  mm 

Theorem:  If  x  =  St  yields  0(£)  =  min  0(x)  and  if  (by  good  luck) 
f ^  ( it: )  =  0  then  x  solves  the  original  problem. 

For  inequality  constrained  problems  we  have  an  analogous  statement. 

We  make  the  obvious  change  and,  in  addition  require 

Xifi(Jt)  =  0  ,  A  >  0  . 

Lagrange's  great  contribution  was  to  make  "the  good  luck"  happen  by 
treating  the  constants  as  unknowns  and  choosing  their  values  by  back  sub¬ 
stitution  so  as  to  satisfy  the  restrictions.  However,  in  certain  applica- 
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tions,  like  linear  programming,  the  Lagrange  approach  degenerates  in  an 
embarassing  manner  that  does  not  make  it  a  practical  approach.  The  basic 
theorem  for  inequalities  constraints  is  due  to  Fritz  John.  Assuming  f^(x) 
are  continuous  and  differentiable  in  the  neighborhood  of  an  extremal  so¬ 
lution,  he  showed  that  the  required  multipliers  exist  if  a  multiplier  X0 
for  the  f  'q ( x )  function  is  included  also.  Kuhn-Tucker,  assuming  conditions 
that  rule  out  such  things  as  cusps,  proved  that  a  set  of  multipliers  ex¬ 
ist  with  XQ  =  1  . 

We  shall  make  a  trivial  extension  of  the  Lagrangian  approach  that 
will  have  important  implications  for  applications. 


First  Idea:  Use  a  Partial  Set  of  Lagrange  Multipliers 


Min  fQ(x) 

fx(x)  <0  1  Xl 

f  (x)  <  0  :  X2 

- ^-partition 


f  (x)  <  0 

fu(x)  <_  0  sub-problem 

0(x)  =  fQ( x)  +  X^f^(x)  +  X2f2(x) 


Theorem :  If  x  =  St  yields  Min  0(x)  subject  to  f^x)  i  0  ,  f^(x)  <  0 
and  if  (by  good  luck)  f^(£)  £  0  ,  f ^(St)  <0  ,  then  St  solves  the  original 
problem. 

Lagrange's  idea  was  to  make  the  "good  luck"  happen 
treat  X.  unknown  constants 

l 

find  St  =  X(X  X,X  ) 

choose  X^,X2  so  that  1  0  ,  f2(£)  <0 

For  inequality  systems,  it  is  usually  difficult  to  treat  £  analytically  as 


a  function  of  X  .  To  get  around  this  difficulty,  we  try  an  alternate  ap¬ 
proach. 

Second  Idea:  Start  with  a  guess.  For  X,  say  X  =  X° ,  examine 
x  =  ,  the  solution  to  the  sub-problem.  Use  it  to  obtain  a  better  guess 

for  X  =  X^  ;  iterate. 


Example :  Find  x  >  0  ,  Min  fQ(x)  : 


Guess  X^: 


Sub- 

Problem 


clxl  +  C2X2  +  °3x3  +  Vl  +  C5X5  *  f0(x) 

allxl  +  a12x2  +  a13x3  +  al4x4  +  a15x5  -  b2 
- 4-partition 


'21X1  +  a22X2 

<  b 

31xl  +  a32x2 

<  b 

al*3x3  +  altl*xl» 

<  b 

a53X3  +  a5Ux4 

<  b 

+  a65x5  -  b6 

01(x1,x2)  +  02(x3,x1()  +  03(x-)=  0(x) 


Subproblem:  Decomposes  into  3  separate  minimization  problems.  Hence 

the  term:  "Decomposition  Principle". 


The  decomposition  algorithm  may  be  described  by  a  chicken-and- 

egg  process.  The  "egg"  part  begins  with  an  estimated  value  for 
k  k 

X1  =  X£  and  X^  =  X^  .  This  defines  a  Lagrangian  problem  with  partial 

v 

constraints.  We  assume  we  have  some  way  to  solve  this  and  that  x  =  x'  . 
Now  x^  is  used  to  initiate  the  "chicken"  part  by  forming  the  vector 
[fQ(xk)  ,  fx(xk)  ,  f2(xk)  ,  l]  .  Earlier  generations  of  chickens  and 
eggs  have  already  produced  the  other  columns  shown  below  in  the  "Master 
Linear  Program".  A  convex  combination  of  these  columns  is  sought  such 


that  their  weighted  average,  component  by  component,  is  non-positive  and 
the  first  component  is  minimal.  This  is  a  linear  program.  The  Lagrange 
Multipliers  associated  with  this  program  are  called  dual  variables. 

Their  optimal  values,  denoted  by  (l,  X^,  X2,A  )  =  (l ,X^+^,X^+1,Ak  1), 
is  now  used  to  initiate  the  next  Lagrangian  problem. 


Master  Linear  Program:  Find  Min  z  and  w^  >_  0  ;  optimal  dual  variables 

(1,  X  ,  X2,  A)  «  (1,  X**1,  Xk+1,  Ak+1)  . 


y*°> 


f0(xl) 


z  ♦  f.  (x  )  w  ♦  f,  (xA)  w.  +  . 


;2(*°) 


[  1  ] 


.f2(x2). 


[  1  ] 


[  1  ]  =  [1] 


The  proof  that  the  iterative  process  converges  assumes  that  the 
functions  f^(x)  are  convex.  Applied  to  the  earlier  referenced  large 
scale  problem,  the  Master  Program  involved  about  300  equations.  The  re¬ 
mainder,  treated  as  one  subproblem,  had  30,000  mutually  exclusive  equations 
and  therefore  would  break  up  into  30,000  independent  problems.  Applied 
to  calculus  of  variations  type  problems,  such  as  the  optimal  linear 
control  processes  considered  by  Pontrayagin,  Bellman,  Zadeh  and  others, 
this  approach  can  lead  to  a  constructive  solution  technique. 

Generally  speaking,  Lagrange  Multipliers  are  treated  in  the 
literature  as  just  a  device.  Courant  describes  it  as  an  "elegant  device" 
for  getting  around  eliminating  dependent  variables  and  then  solving  a 
simple  minimization  problem  without  restrictions.  Recent  research,  however, 
reveals  that  it  has  a  deeper  meaning. 
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